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ABSTRACT 

This paper deals with development of logical structures for e-leaming evaluation. Evaluation is a complex task into which 
many different groups of people are involved. As a rule these groups have different understanding and varying 
expectations on e-leaming evaluation. Using logical structures for e-learning evaluation we can join the different 
expectations of groups and can evaluate quality of e-learning with multidimensional targets. 

Key targets form the base for a logical evaluation structure. The evaluation is successful or positive only if all key targets 
were achieved. If only one key target was not achieved the whole process has missed its target. 

As a rule key targets can be achieved on different ways. This can be mapped by corresponding sub targets which form the 
inner logical stmcture of key targets. By this method very flexible target stmctures can be modeled. If consensus has been 
reached on the logical target structure of an e-leaming project further evaluation can be performed based on adapted 
checklists and the use of measure theoretical methods as that has been presented in previous papers of authors. 
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1. INTRODUCTION 

Evaluation is playing a very important role during development, implementation and realization of an 
e-learning framework. Educational evaluation models and approaches were developed by several authors: 
D.L.Kirkpatrick and J.L.Kirkpatrick (2006), and especially for e-learning evaluation to Stufflebeam (1972), 
Khan (2004), Ehlers et al. (2006), Colace et al. (2006), Lam and McN aught (2008) and Ruhe and Zumbo 
(2009). 

However, the groups which are involved into an e-learning framework: pedagogues, administrators, 
educators, multimedia designers, managers, stakeholders, decision makers, tutors, and learners have 
distinguished ideas and expectations on evaluation. 

Moreover, e-learning is as a rule a work, time and cost intensive project which needs development and 
qualification from one implementation to the next. It is a team work result of different groups, experts or 
partners. If we try to measure the quality of e-leaming we have to take into consideration the interests and 
expectations of all involved groups. Besides the groups mentioned above this can be moreover educational 
institutions and financial backers, for instance. 

Eigure 1 emphasizes that the interested groups in e-learning evaluation can have different expectations 
and distinguished targets. But all these groups have one common main target. They want to measure anyhow 
the quality of e-learning or want to know how successfully an e-learning course is running. 
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Figure 1 . E-learning and evaluation 

To create and develop e-learning courses different environments have to work together (Khan, 2004): 

• Pedagogical environment 

• Interface Design environment 

• Management environment 

• Ethical environment 

• Resource support environment 

• Institutional environment 

• Technological environment 

By means of logical structures we can evaluate the quality of an e-leaming course not only in sense of 
expectations of involved different groups we can measure the quality of an e-learning course for each 
environment, too. 


2. LOGICAL STRUCTURES OF KEY AND SUB TARGETS 

By our model it becomes possible to evaluate for all involved groups how the corresponding group has 
achieved its target. Based on an adapted logical structure we can see how successful the components of an e- 
learning process are running, where we have to spend more attention and what should be improved for the 
next round. A further advantage of our model is we can consider very flexible logical structures, if necessary. 
Simultaneously it becomes possible to evaluate in a consistent manner how special single targets were 
achieved. 

The formal details of this approach are presented by Uranchimeg, Hardt (2011). Key targets are involved 
into a logical structure via serial schemes. A corresponding main target can be achieved only if each key 
target of logical structure is achieved. If only one of key targets is missed the corresponding main target is 
missed, too. 

Sub targets of key target are visualized by a parallel schemes (see Figure 2). That means if at least one of 
the sub targets is achieved we will consider the corresponding key target as achieved. A key target can 
consists of several sub targets. But to extensive parallel structures should be avoided. In this case we can get 
high evaluation values for a key target even in cases when the evaluation values for the sub targets are 
uniformly small. This should be noted during the design of a logical structure. 

After definition of logical structure of an evaluation target the evaluation scores can be computed by 
means of the calculation rules presented in Uranchimeg, Hardt (2012) based on the observation data obtained 
by adapted checklists. 
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Figure 2. Visualization of logical structure 

For instance, the score that an e-learning course whose logical target structure is defined as it is 
demonstrated in Figure 2 is calculated by 


=n(i-n^ 


=1 ;=1 


7=1 


Here denotes q*ij = Q*(Aij^ for i = 1, = 1, the estimation value for the score that the single 

target Aij has reached its target. These estimation values are determined from the checklist data. It holds 
0 < Q*{C) < 1 as well as 0 < qij < 1. A value Q*{C) = 1 means the main targets of a target structure C 
have been reached completely, Q*{C) = 0 means the targets were missed. For the details we refer again to 
Uranchimeg, Hardt (201 1). 
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Figure 3. Logical structure of different groups 
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The e-learning evaluation process consists of eight steps (see Figure 3): 1) definition of key targets, 2) 
definition of sub targets, 3) confirmation of evaluation targets, 4) creation of checklist, 5) acceptance of 
checklist, 6) data collection according checklist, 7) data processing, 8) interpretation of evaluation results and 
outcomes. Some steps are subjective oriented (characterized in Figure 3 by ‘S’), some are more objective 
oriented (characterized in Figure 3 by ‘O’). Subjective steps should be repeated until agreement is reached 
between stakeholders or group leaders which are involved into e-leaming process. 


3. CONCLUSION 

The structure oriented scoring model is an alternative to the frequently used linear evaluation models. By 
concentration to the logical structure of target of an e-learning process and calculation of the score by 
application of the corresponding rules of general measure theory we don’t need the more or less subjective 
weight factors which are used for linear models. 

The presented model offers the possibility to consider parallel and consistent the expectations of the 
different groups which are involved into development and implementation of an e-learning process. Hence, 
the model can be applied for global evaluation of an e-leaming process as well as an evaluation of embedded 
processes. Via the consideration of group relevant logical sub structures it becomes possible to involve and to 
integrate all interested groups into an evaluation process. The corresponding data can obtained based one a 
general checklist where the data are processed according to considered logical stractures of evaluation 
targets. This can avoid frequently observed conflicts at determination of priorities for evaluation. Using one 
checklist we can collect data from learners about all environments. 

The use of model in context of formative evaluation can help the institutions to recognize weaknesses of 
an e-learning process or to monitor running processes. 

Moreover, the structure oriented scoring model can be used in more general sense for stractural 
evaluation and monitoring of functionality of general processes with a logically describable function target. 
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